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Requirements  Engineering  Challenge: 

Applying  Measurement  &  Analysis  to  Qualitative  Problems 


Users  &  customers  often  have  unrealistic  performance  expectations.* 

To  meet  this  challenge  in  military  acquisition,  it  is  not  enough  that 

•  program  managers  begin  to  collect  unbiased  data  to  analyze  project  costs  &  projections. 

•  programs  perform  technology  maturation  activities,  competitive  prototyping  or 
Preliminary  Design  Reviews  (PDRs). 

Most  importantly, 

•  Mutual  understanding  of  capabilities  in  context  &  what  it  takes  to  enable  them 
need  to  be  established  before  a  project  is  initiated  among  those  who 

-  determine  what  capabilities  are  needed 

-  write  requirement  specifications 

-  acquire  the  systems  that  meet  specifications, 

so  that  appropriate  measures  of  performance  &  other  quality  attributes  of 
the  capabilities  &  their  enabling  systems  can  be  determined  and  aligned. 

*  OPINION  :  Reforms  for  the  Department  of  Defense,  by  Sen.  Carl  Levin  (D-MI),  Niles  Star,  Michigan,  03/11/2009  re  the  Levin-McCain  2009  Weapons  Systems  Acquisition  Reform  Bill 
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A  Promising  Solution 


A  key  to  understanding  quality  attributes  is  handling  qualitative  data, 
which  in  large  part  is  language  data,  &  making  it  quantitative. 

Two  complementary  language  data  techniques  are  being  used 

•  KJ  Analysis 

—  Structured  methods  for  eliciting  &  clarifying/interpreting  semantic  meaning  of 
textual  information 

—  That  automated  text  analysis  simply  does  not  have 

•  Semi-automated  content  analysis  based  on  automated  text  analysis 

—  Enables  more  input  from  more  stakeholders  and  identifies  concepts  in  common 
that  enables  consistent  applications  of  KJ  across  time  &  sites 

Merging  the  two  allows  understanding  of  language  data  crucial  for 
requirements  &  their  measured  validation. 
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Language  Data:  “The  Other  Data  of  Measurement 
&  Analysis”*  for  Requirements  Engineering 


ore  there  essential 
planning  capabilities 
not  enabled  by  military 
systems? 


Thinking: 

Distilling 

Understanding 


Hypothesis 

(Guess) 


Test 


/  V  Generalization 


Validation  & 
Verification 


Abstract 


Confirmation 

or  Falsification  Experience. 

Discovering 

Gathering 


Data 


*  Adopted  from  David  Hallowell,  Language  Data:  The  ‘Othei 
Data’  of  Six  Sigma:  Part  1  of  2,” 

http://software.isixsiqma.com/librarv/content/c040303b.asp 
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KJ  -  a  Method  for  Collaborative  Processing  of 

Language  Data 


KJ  (named  after  Jiro  Kawakita)  is 
a  method  for  transmuting  tacit 
knowledge  into  explicit  and  more 
&  more  objective  statements 


Structured 

Interviews 


Emotion 


“I  hale  the 
new  office 
layout" 


"There's  more 
background  noise 
-and  itfie  to 
prevent  people 
from  popping  in 
to  rry  office  wti 
interruptions  - 
that  can  take  up  a 
good  20%  of  my 
day." 


...  and  refining  a  collection  of 
these  statements  into  a  2-D 
patterned  representation  (KJ 
template)  that  conveys  lots  of 
information  and  related 
thought  processes  for  those 
communicating 
asynchronously. 


1391111 

sT 

...  .1  r- — .  .  —  •  TV  * 


► 


i».. 


Judgment 


‘IT  did  a  poor 
job  wth  the 
upgrade." 


I 

better 

. k . 

“Transactions  posted 
with  a  mix  of  ihe  old 
and  new  software 
corrupted  the  orders 
database  -  requiring 
4  people  to  spend  a 
week  straightening  it 
out” 


Inference 


“We  need  a 
purchasing 
speciafston 
site” 

I 

better 

. k . 


“About  20%  of  the 
project  slip  was 
due  to  lime  in 
getting  paperwork 
in  and  out  of 
corporate 
purchasing." 


Unclear  Measures 


“We  have  a  lot  of  new 
people  that  need  to 
come  up  to  speed 
soon.” 

I 

better 

_ L. _ 

‘We  have  40  new 
account  reps  that 
need  to  be  able  to 
create  and  manage 
customer  orders 
and  deliveries  by 
the  start  of  next 
quarter.” 


Adopted  from  David  Hallowell,  Language  Data:  The  ‘Other  Data’  of  Six  Sigma:  Part  s  1  &  2,” 

http://www.isixsiqma.com/librarv/content/cQ40303b.asp 
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Distilling  Meaning  in  Language  Data: 

Benefits  of  Combining  KJ  with  Automated  Content  Analysis 


Content  analysis  (CA)  methods  to  analyze  large  bodies  of  textual  documentation 
have  existed  for  over  seventy  years,  but  until  recently  were  predominantly  manual. 

•  They  were  used  during  World  War  II  to  predict  the  bombing  of  London  by  analyzing 
Joseph  Goebbels’  speeches. 

•  However,  humans  cannot  read  &  digest  all  the  documents  &  recognize  all  the  patterns 
that  machines  are  getting  increasingly  better  at  doing. 

•  Yet  humans  must  interpret  the  meaning,  or  lack  thereof,  of  what  machines  can  find. 

Benefits  of  Combining  CA  &  KJ 

•  Automated  text  analysis  can  process  a  much  richer  body  of  information  &  provide 
common  concepts  for  KJ  participants  to  establish  consistency  from  group  to  group. 

•  KJ  analysis  provides  a  framework  for  collaborative  interpretation  of  concept  maps 
produced  by  CA. 

Upshot:  The  combination  is  a  hybrid  that  alters  both  CA  &  KJ  but  that  generates 
insights  that  neither  could  produce  alone. 
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Insuring  Consistency  &  Completeness 
Across  KJ  Settings  is  a  Problem 


To  date,  KJ  analysis  has  been  applied  in  small,  face-to-face  one-day  workshops 
where  both  customer  and  technologist  points  of  view  are  represented. 

One  worry  is  that  results  might  be  quite  different  if  different  representatives 
were  assembled  on  a  different  day. 

An  automated  language  or  content  analysis  approach  is  being  developed  to 
supplement  the  KJ  method  so  that 

•  a  significantly  larger  group  of  individuals  in  geographically  disparate  locations  can 
participate  asynchronously. 

•  additional  textual  information  is  captured  beyond  that  in  KJ  face-to-face  sessions: 

-  documentation  in  the  form  of  requirement  descriptions  &  specifications 

-  problem  and  defect  report  databases 

-  open  text  surveys 

•  there  is  more  confidence  that  results  are  complete  and  repeatable. 
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Content  Analysis  &  Concept  Maps  - 
a  Language  Data  Computational  Processing  Method 


CA  uses  automated  text  analysis  tools  to  identify  recurring  concepts  &  clusters  of  concepts: 

•  Concepts  are  synonyms  of  strongly  related  co-occurring  terms 

—  constituted  in  automatically  generated  affinity  lists 

—  named  by  most  representative  term  in  affinity  list 

•  Concept  Clusters  are  collections  concepts  of  similar 
co-occurrence  patterns 


'T 


—  more  strongly  related  to  each  other  than  to  concepts  in  other  clusters 

—  named  by  automatic  selection  of  the  concept  most  strongly  related  to 
other  concepts  in  the  cluster 


Concept  Clusters  are  represented  graphically  as  Venn  diagrams 

•  Concept  names  labeling  dots  are  in  concept  clusters  represented  as  circles 

•  dots  can  be  linked  by  lines  whose  brightness  represents  frequency  of  co-occurrence 

•  dots  can  appear  in  the  overlap  of  two  (or  more)  circles 

•  circle  size  does  not  always  indicate  importance  since  circles  can  be  sparsely  populated 


An  Innovative  Requirements  Solution:  Marrying 

Software  Engineering  Institute 

©  2009  Carnegie  Mellon  University 


Carnegie  Mellon 


Six  Sigma  KJ  Analysis  with  Automation  for  Text 
Analysis  and  Collaboration  |  SEPG  2009  North 
America 


A  CMMI-ACQ  Example:  Concept  Map  Clusters 
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components 

available 


Trapping 


Software  Engineering  Institute  CamegieMellon 


maintain 


An  Innovative  Requirements  Solution:  Marrying 
Six  Sigma  KJ  Analysis  with  Automation  for  Text 
Analysis  and  Collaboration  |  SEPG  2009  North 
America 

©  2009  Carnegie  Mellon  University 


Handling  Language  Data  Numerically 


The  most  frequent  CMMI-ACQ 
concepts  are  listed  at  the  left. 

The  absolute  count  is  the  number 
of  text  blocks  where  a  concept 
occurs  -  highest  count  set  at  100%. 

The  relative  count  is  the  percentage 
of  text  blocks  concepts  occur  in. 

Not  surprisingly  for  a  process 
model,  conceptual  traces  of 
process  are  found  in  the  most 
CMMI-ACQ  text  blocks. 

Project  and  organization  are  the 

next  most  significant  thematic 
concepts. 

These  are  followed  by  product  and 
then  supplier  all  of  which  are 
important  to  the  points  made 
previously 

All  are  in  the  top  10%  of  concepts 
appearing  in  concept  maps  that 
follow. 


Command  &  Control 


Organize 

Establish  &  maintain  unity  of  effort  w/  mission  partners 

Develop  Trust 

Estab  &  Cultivate  Rel  w  Msn  Partners 
Estab  &  Cultivate  Rel  w  Partner  Orgs 
Structure  organization  to  mission 
Define  structure 
Assess  Staff  Capabilities 
Delegate  Authority 
Identify  Capabilities  Needed 
Integrate  Capabilities 
Estab  Commanders'  Expectations 
Foster  organizational  collaboration 


Estab  Collaboration  Poliaes 
Estab  Collaborative  Procedures 

Understand 

Organize  Information 

bevelup  Ki  luwleUye  ai  iU  Situational  Awareness 
Share  Knowledge  and  Situathpal  Awareness 
Planning 

Analyze  problem 
Analyze  Guidance 
Review  Rule  Set 
Review  Situation 
Determine  Need  for  Action 
Prepare  Estimates 
Apply  situational  understanding 
Assess  Available  Capabilities 
Evaluate  Environment 
Determine  Vulnerabilities 
Determine  Opportunities 
Develop  strategy 
Determine  Force  Readiness 
Determine  Resources 
Adapt  Strategy 
Align  Strategy 
Develop  Assumptions 
Develop  Objectives 
Determine  End  State 
Review  Existing  Plans 
Develop  courses  of  action 
Understand  Objectives 
ne.vdnp  Qitinns 


Decide 

Manage  risk 

Validate  Targets 
Formulate  Crisis  Assessment 
Provide  Friendly  Force  Combat  Identification 
Direct  Consequence  Management 
Select  actions 
Select  course  of  action 
Select  Plan 
Ter  min  ate 
Establish  rule  sets 
Establish  intent  and  guidance 
Establish  Priorities 
Establish  Standards 
Establish  Rule  Sets 
Intuit 

Recognize  Key  Triggers 
Modify  Actions 

Direct 

Communicate  intent  and  guidance 

Issue  Estimates 
Issue  Priorities 
Issue  Rule  Sets 
Provide  CONOPS 

Task 

Synchronize  Operations 
Synchronize  Execution  across  Phases 
Issue  Plans 

- Ioouo  Or  dor  o - 


Capability 

Area 


Establish  Selection  Criteria 
Analyze  courses  of  action 
War  game  oourses  of  actions 
Compare  courses  of  actions 


Establish  metrics 

Establish  Performance  Measures 
Establish  Effectiveness  Measures 

Monitor 

Assess  compliance  with  guidance 

Assess  Employment  of  Forces 
Assess  Manner  of  Employment 
Assess  effects 
Assess  Battle  Damage 
Assess  Effects  of  Deception  Plan 
Assess  Munitions  Effects 
Assess  Performance  J 

- Assess  Ru"&  lyayei  i  iui  it  Ruquii  urfient 


Assess  Operational  Effects  of  Strategic  Communications 

Assess  achievement  of  objectives 
Assess  guidance 


Operational 

Capabilities 


Army  BC 
Functional 
Concept  (TP 
525-3.3 
2007) 


Army  C2 
Doctrine 
(FM  6-0 
2003) 


Planning  Aviation 
Missions  (PAM) 
Desired  Functions 
Description  (DFD) 
2007 


Combining  KJ  and  CA 
An  Example: 

Starting  with  military 
capability  areas  & 
corresponding 
documentation 

and  documentation  of 
systems  that  may  enable 
essential  capabilities, 

formulate  a  Theme 
Question: 


Are  there  essential  planning 
capabilities  not  enabled  by 
military  systems? 


Army  Aviation 
ISP  for  Mission 
Planning 


System 

Functions 
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Combining  CA  with  KJ  2 


Are  there 
essential 
planning 
capabilities 
not  enabled  by 
military 
systems? 


Automated  Content  Analysis 


Statements  &  SubThemes  3.1  &  3.2  were  formulated  In  discussion  with 
representatives  from  PEO  Aviation.  They  are  still  preliminary. 

More  discussions  with  more  Aviation  groups  is  being  planned  using  a  mix 
of  KJ  and  CA  methods  both  synchronouslyand  asynchronously. 


evolution  of  Aviation 
planning  capabilities 
&  Hi  er  quality 
attributes 


-=Q. 


:  - 


r> 


>£bon:  ifnere  are  essential  planning  capa 
'by  the  intemperation  of  Aviation  mission  p 
sharing  systems,  what  shotrid  be  done? 


SU bill efne3-l:> providing  acapabily  liie 
adjuring  plans  n-4tgM  e  an  abortive.  plaming  & 
analyst  b  neafcd  across  progrsns  respnnsibte 
forthemiitipt  systems  enabing the  capability 


laeuuon  execution  as  atonnrf  idly 


FPTitaMBSfiS 


^flivty&iiiHi^»aiMaya~EqiMtymna^s 
fbrhntti  nmmbng  Mid  saluitiUi^nffi 


realignment 


Theme  Question:  Given  that  there  are  essential  planning 
capability  gaps  not  covered  by  the  interoperation  of  Aviation 
mission  planning  and  information  sharing  systems,  what  should 
bedone7 


Tl»eme3:  Establish 
ptocesses-ibr 


Themed  (DFDft  ISP):  AipHy&flexaritty 


The  conditional  of  the  theme  question  appears  to  be  true, 
so  what  should  be  done? 
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Iterations  =  1000 
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critical 
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superiority 
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actions 


integrated 


enemy 


mission 


initiative 


order 


require  exercise 


forces 
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environment 
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systems 
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concept 
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Army  BC  Functional 
Concept  (TP  525-3.3 
2007)  -  planning, 
reframing,  execution, 
data,  METT-TC,  time, 
information, 
interoperability  & 
agility 


operation^ 


training 


retrying 


assessing 


,  -.tent 
execution  IfHiBt  ^ 


rocess 


level 


function 


Control 


network 


interoperability 
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Army  Battle  Command  -  Continuous  (Re)Planning 

(TP  525-3.3  BC  Functional  Concept) 


METT-TC 


PLAN 


Assess 


From  FM  3-0 


At  the  operational  and  strategic  levels,  the 
commander  frames  the  existing 
conditions  by  interrelating  PMESSII-PT 
factors  (Politics,  Military,  Economic, 

Social,  Information,  Infrastructure, 

Physical  &  Time). 

At  the  tactical  level,  commanders 
consider  METT-TC  factors  (Mission, 
Enemy,  Terrain  &  weather,  Troops  & 
support,  Time  available,  Civil 
considerations)  wrt  what  they  are  learning 
from  accumulating  PMESII-PT  information 
covering  both  friendly  forces  &  enemy 
forces. 


After  initial  planning,  framing  is  referred  to 

the  enemy.”  (Moltke  in  FM  6-0)  as  reframing- 
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Elements  of  a  KJ  Template 


Theme  Question:  If  there  are  essential  planning  capability  gaps 
not  covered  by  the  interoperation  of  Aviation  mission  planning  and 
information  sharing  systems,  what  should  be  done? 


Themes  abstract¬ 
meaning  from  groups  of 
subthemes 

SubThemes  abstract 
meaning  from  groups  of 
statements 


Key  data  in  KJ  are 
associated  concepts 
formulated  in 
statements 


Statements  and  subthemes 
are  grouped 


Themel  (TP525-3.3  &  FM  6-0):  An 

essential  planning  capability  is  reframing  or' 
adjusting  plans  during  mission  execution 
requires  agility  dependent  on  interoperability 


5-0:  Plan 


^SubThemel  .1  TP525-3.3: 
Plan  reframing  during  missions 
takes  leader  agility  that  is 
dependent  on  interoperability 


lanning  reframing  based  on 
data,  information  &  METT-TC 
during  execution  requires 
leaders  agility 


Interoperability  applies  to  data, 
planning,  information  &  agility 


Agility  &  interoperability  are 

mutually  dependent  quality 
attributes  but  not  operationalized 


SubThemel  .2  FM  6 
adjustment  during  missions 
takes  flexibility  in  leading 
dependent  on  interoperability 


No  plan  survives  intact 
contact  is  made 


Flexible  leaders  &  staffs 
must  understand  &  adjust 
to  changing  situations  to 
alter  plans 


Mutual  dependence  of 
flexibility  &  interoperability 

not  explicit  &  neither  quality 
attribute  is  operationalized 


Concepts 
used  in 
themes, 
subthemes  & 
statements 
derived  from 
content 
analysis  are 
in  bold 
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Concepts  Used  in  KJ  Templates  Derived  from 

Concept  Maps 


Theme  Question:  If  there  are  essential  planning  capability  gaps 
not  covered  by  the  interoperation  of  Aviation  mission  planning  and 
information  sharing  systems,  what  should  be  done? 

Themel  FM  6-0):  An 

essenfalplann wig  capabBy  or 

plans  duiig  mission 

requinos  agility  dependent  on  inftgropeiatbiiify 


I.11P525  3J: 


ord^t. 


Futur».Mo*jl.r.Fo"ir 


aifFta  BfiaHibsedon 

> 

/r  V  -X 

mnbSSmale 

\ 

|  Mws  |TI/ 

Miiual  dependence  of  \ 

aiiiHyirii  nijri 

s 

&Hfcnn»ri»atyare 

not  explnt&  nesthc^tanHy 
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Misalignment  between  TP  525-3. 3/FM  6-0  & 

PAM  DFD/ISP 


Theme  Question :  Given  that  there  are  essential  planning  capability 
gaps  not  covered  by  the  interoperation  of  Aviation  mission  planning 
and  information  sharing  systems ,  what  should  be  done? 


Themel  (TP525-3.3  &  F  M  6-0):  An 
essential  planning  capability  is  refraining  or 
adjusting  plans  during  mission  execution 
requires  agBly  dependent  on  inlbe  rope  2 1  i  s  a  I  b^i  il  |  itl^y 


misalignment 


SubThemel.l  TP525-3.3: 

Plan  refrawitfl  diving  i 
takes  leader  agBty  fiat  is 
dependent  on  i 


Planning  refraining  based  on 
data,  information  &METT-TC 
duing  execution  reqiires 
leaders  agility 


Interoperability  applies  to  data, 
planning,  information  &  agility 


Ag  iltty  &  interoperability  are 
mutually  dependentquaity 
attributes  butnotoper^iionai^cl 


SubThemel  J2.  FM  6-0:  Plan 
:  diving  i 

I 

dependent  on 


No  plan  sivvives  irt act  once 
contact  is  made 


Flexble  leaders  &  staffs 
must  understand  &  adjust 
to  changing  situations  to 
atter  plans ™ 


MUtiiaJ  dependence  of 
flexibility  &  interoperability 
not  expl  iat  &  neither  q  uafity 
^tribute  is  o  perdional 


Theme2:  (DFD  &  ISP):  Agiilii%  &  flexibility 
are  not  specified  &  interoperability  iis  not 
sufficiently  specified  nor  operationalized; 
adjusting  plans  in  (flight  not  really  considered 


SubTheme2.1 :  The 


DFD  does  not  explicitiy 
address  plan  adjustment 
during  execution  norspecify 
agility  &  interoperabHity 


The  system  is  moregearedfo 
loading  Data  before  flight  & 
using  itto  execute  a  plan  ■ 


Adjusting  plans  takes  place 
before  loading  datainfo 
aircraft  systems ~~ 


Interoperability  is  included  as 
an  objective  but  only  ip  to  the 
syntactic  level  — - 


V  V 


SubTheme2.2:  The 
ISP  requires  interoperability 
at  a  syntactic  level,  but  does 
not  define  interoperability  at 
different  levels 


The  conditional  of  the  theme  question  appears  to  be  true, 

so  what  should  be  done? 
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Given  the  Misalignment,  What  Should  be  Done? 


Theme  Question:  Given  that  there  are  essential  planning 


capability  gaps  not  covered  by  the  interoperation  of  Aviation 
mission  planning  and  information  sharing  systems,  what  should 


be  done? 


Theme3:  Estabfish 
processes  loir 
evolution  ofAviation 
planning  capabilities 
&  their  quality 
attributes 


S ubTheme3 .1 :  tf  providing  a  capability  lice 
adjusting  plans  in-flight  is  an  objective,  planning  & 
analysis  is  needed  across  programs  responsible 
for  the  multple  systems  enabling  the  capability 


Sensor,  p  laming  &  in  formation  shamg 
systems  are  mutually  responsble  for  enabbig 
plan  adjustment dunng  I 


There  are  technical ,  tactical  &  himan  li  rotations 
on  the  extent  to  which  plamrig&  missions  can 
be  refrained  both  pre&duingflic^t  ^ 


Mittiple  systems  have  to  interoperate 
throucJioUttheirevolUtion 


SubTheme3.2:  Establish  a  process 
for  defining,  operationalizing  &  aligning 
quality  attrinites  lice  agility  &  interoperability 
across  combat  forces  and  systems 


A  preliminary  answer  is  provided  in  Statements  &  SubThemes  3.1  &  3.2  formulated  in 
discussion  with  a  few  representatives  from  PEO  Aviation. 

Elaboration,  confirmation  &  buy-in  has  to  be  achieved  with  many  more  stakeholders. 

More  discussions  with  more  Aviation  groups  is  being  planned  using  a  mix  of  KJ  and  CA 
methods  both  synchronously  and  asynchronously. 
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Relationships  of  all  Three  Themes 


Theme  Question:  Given  that  there  are  essential  planning  capability 
gaps  not  covered  by  the  interoperation  of  Aviation  mission  planning 
and  information  sharing  systems,  what  should  be  done? 


Themel  (TP525-3.3  &  FM  6-0):  An 
essential  planning  capability  is  reframing  or 
adjusting  plans  duiing  mission  execufloira 
taking  agility  dependent  on  interoperability 


Theme2:  fOFO  &  ISP):  Agility  &  flexibility 
are  not  specified  &  interoperability  is  not 
sufficiently  specified  noroperationalized; 
adjusting  plans  in-flight  not  really  considered 


1.1  TP525-33: 
j  dning  i 
takes  leader  agiity  that  is 
depen  dent  on  mteropeialimy 


-2  FM  6-0:  Fla 
l  dLirig  i 

flityi 

dependent  on  ntcropt 


/f= 


based  on 
&METT-TC 
during  execution  requires 


rnitualy  depended  quaky 
attributes  but  n  of  o  pefatxmafiaed 


Mutual  dependence  of 
fleAtidy  &interoperabdty 
not  explicit  &n  either  quaky 


realignment 

T 


SubTheme2.1 = The 
DFD  does  not  explicitly 
address  plan  ac^usfcnent 
during  execution  nor  specify 
agility  &  interoperability 


SubTtieme2-2z  The 
ISP  requires  interoperability 
at  a  syntactic  level,  but  does 
not  define  interoperability  at 
different  levels 


1  The  system  is  more  geared  ta 

loading  Data  before  fhg  lit  & 
1  using  itto  execute  a  plan  “ 

1  Adjustmg  plan  stakes  pfact 
before  loafing  data  into 

I  aa craft  systems  — 

biter  opetHlxfety  is  inckided  as 
an  objecfive  but  only  uj>  to  the 
syntactic  level  — ■ 

^ _ s 

p  ra  v  lied  by  mterop  eratidtj 
ftMictionsthat  support 
traiderofdatain  various 


0, 1  or2are  used  but  not 
defined.  Id  alone 
operabonakzed  with  measures 
specified 


SubTheme3.1  2  If  providing  a  capability  like 


SubTtieme3*2=  Estabish  a  process 


Scaling  Up  the  Process 


Automated 


Human 
Interactive 


1  - 

[ 

M 

■ 

9 

M, 

Multiple  face-to- 
face  meetings 
where 

statements  are 


formulated  & 
.  collected 

llfSSaiis 


lillli 

^  ... _ 

iillRiil  3IiiliI£2. 

'4  "z.  -  5  -» 

s  m  ^ 


I  *  :r 

si  1  | 


wM 

-  v 

.... 


•  .  .  .  •  ;  ' 

*33Ms3 


Iliiai 

jpsfi 


*-*  Jfc“r 

i_ j 

1  -1 

1  -J 

SI 

^Jj 

IS 

=J - \r 

Lt 

j|i  J  1 - 

•  1  -J 

i= 

INN 

,  i  B.  t 

Jl;fT 

i= 

r  t  3 

1  1 - J 

-  J - =1 
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Concluding  Thoughts 


What  has  been  shown  is  the  potential  for  CA  to 

•  uncover  misalignments  among  documents  describing  capabilities  &  systems 

•  identify  quality  attributes  in  these  documents  that  need  to  be  better  defined  & 
operationalized 

These  results  can  be  used  in  KJ  analyses  in  the  form  of  common  concepts  that  can 
be  combined  in  statements,  subthemes  &  themes  in 

•  multiple  face-to-face  interview  probing 

•  collaborative  interpretation  of  concept  maps  derived  from  different  documents  leading 
to  determinations  of  their  alignment  or  misalignment. 

So  far  this  combined  use  of  CA  &  KJ  has  shown  promise  when  used  informally  with 
a  few  representatives  of  military  organizations. 

The  next  step  is  to  refine  the  emerging  process  with  greater  numbers  of  people 
and  documents  in  multiple  settings. 
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Thank  you  for  your  attention! 


For  further  information, 
please  contact: 

Ira  A.  Monarch 

iam@sei.cmu.edu 

1.412.268.7070 

Dennis  R.  Goldenson 

dg@sei.cmu.edu 

1.412.268.8506 

Robert  W.  Stoddard  II 

rws@sei.cmu.edu 

1.412-268 
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Backup 
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Iterations  =  1000 


experience 


opportunities 


cycle 


M«4pip 


changes 


lead 


state 


situation 


achieve 


exec<H$ftditions 


flexibility 


cla“‘'”'Army  C2  Doctrine  (FM  6-0 

2003)  -plan,  adjustment, 
lead,  flexibility, 
commander,  staff, 


C^a 


decisions 
focus 


knowledge^ 


judg  actions  enemy 

friendly 


time 


situational_understandln§IStment 
events  sv 


analysis 


process 
current 


initiative 


detai^r  interoperability,  execution, 
an  situational_understanding 


trust 

mutual 


i7ort 


exercise 


direct 


leadership 
coricept  decisive 


subordinates  accomplish 

subordinates 


rice 


execution 


comrriunica^i 


rms 


levels  /  risk 
environment 


mission  d 

£y 


octrine 


battle 


M^t-tc 

critical 

available  "—-Sgy£ces 

B^ss  estimates  future  assessment  j 

information  problems 

Ri  information 

(01 


«2 


orders 


effects 
terrain 


forces  nature 
place 


/ 

/  a 


A 


capabilities 

weather 


location  /  authority 

^  command 


i 

X 


major 


operations 


combat 


command 

control 


Ttp 


resources 


data 


rehearsal 


determir 


P*an  preparation  guidance 

tactical 

Trainingground 

echelons  planning 

joint 

organization 

help 

civil 

soldiers 

procedures 

(er  intelligence  communications  r 

tasks 

novement  area 

reconnaissance 

Abes 


commander 


rehearsal 


activities 


duties 


functions 

fire 


assigned 

defense 


interoperability] 


staff 
requirements 


G-6 


support 


team 


personnel 


equipment  Jgfti 

personal  air 
H  individual 


facilities 
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Army  Command  &  Control  Doctrine  (FM  6-0) 


CCIR 


PIR/FFIR 


Adjustment  f 

/ S'  Visions  / 

y^^  ▼  Execution 

V.  .  Plsn  decisions 

1 

COP 

Apply 

Combat 

1 

Situational 

Understanding 

naTt  \ 

js.  Power 

Type 

Decision 

Situation 

Action 

Analysis 


IR:  METT-TC 
(Questions) 

ReWvar'l 

Information 


decisions 


Collectors 


y?  METT-TC 


c 

o 

3 

o 

s 

LU 


Tactical 

Environment 


Collection 


Application  of  the  M  ilitary 
Decision  Making  Process 
(MDMP)  prior  to 
execution  projects 
synchronized  application  f 
of  combat  power. 

During  execution 
variances  due  to 
unanticipated  situations 
disrLpts  synchronization. 


Visualize 
•  **ue  n:a  Guwance 


Describe 

•AppTMRMaCMMttlOfl 

•  Itsue  Irma  cars  mt trt 

•  Cdrt  s>onnng 
Gjturot 

•  Approve  macc« 

•  Rerine  Puming  Guttance 


} 

> 


Direct 

•  Approve  COA 

•  Refine  cars  went  /  CCK 

•  ApfTOw*  OPtAHOPO«0 


> 


Step  1  -  Receipt  of 

■  .  1  WARMO  1 

Step  2  -  Mission 
Analysis 

Mission 

Analysis 

Briefing 

Step  3  -  COA 
Development 

COA  Briefing 

Step  4  -  COA 
Analysis 

sta 
|  EaOm 
War  Garni 
Briefing 
(optional) 

IT 

at—  | 

Step  5  -  COA 
Comparison 

Step  0  -  COA  Approval 

COA  Decision 
Briefing 

Step  7  -  Orders 
Production 

OPORD  Briefing 

r 

C 

0) 

E 


TJ 

< 


MDMP 


Minor  Variances  from  the  Plan 

Operation  proceding  according  to  plan 
Variances  are  within  acceptable  limits. 


Anticipated  Situation 

Operation  encountering  variances  within 
the  limits  for  one  or  more  sequels. 


Unanticipated  Situation- 
Friendly  Success 

Significant  unanticipated  positive  vanances 
result  in  opportunities  to  achieve  the  end 
state  in  ways  that  differ  significantly  from 
the  plan. 


Unanticipated  Situation- 
Enemy  Threat 

Significant,  unanticipated  negative 
vanances  impede  mission  accomplishment. 


Execute  Planned  Actions 

Commander  or  designee  decides 
which  planned  actions  best  meet 
situation  and  directs  their  execution. 
Staff  completes  follow-up  actions 
Decision  may  be  permissive 
FRAGO  not  normally  issued 


Execute  a  Branch  or  Sequel 

Commander  or  staff  review 
branch/sequel  plan. 

Commander  receives  assessments 
and  recommendations  for  modifi¬ 
cations  to  the  plan,  determines  the 
time  available  to  refine  it  and  either 
issues  guidance  for  further  actions  or 
directs  execution  of  a  branch/sequel. 
Staff  issues  FRAGO. 

Staff  completes  follow-up  actions 


Make  an  Adjustment  Decision 

•  Commander  recognizes  threat/ 
opportunity  and  determines  time 
available  for  decisionmaking. 

•  Commander  selects  a  decisionmaking 
method.  If  there  is  not  enough  time  for 
a  complete  MDMP.  the  commander 
directs  the  staff  to  refine  a  single  COA 
or  directs  actions  by  subordinates  to 
counter  the  threat/exploit  opportunity 
and  exercise  initiative  within  the  higher 
commander's  intent 

•  Commander  normally  does  not  attempt 
to  restore  the  plan 

•  Commander  issues  a  verbal  WARNO 
or  FRAGO  to  subordinate 
commanders 

•  Staff  resynchronizes  operation, 

modifies  the  criteria  of  success,  and 
begins  assessing  operations  for 
progress  using  the  new  criteria  of 
succe», _ _ _ | 
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Interpreting  Analysis  of  TP  525-3.3  &  FM  6-0 

Quality  attributes  need  to  be  specified  &  operationalized 


Plan  reframing  during  mission  execution  is  a  form  of  leader  agility  (inference 
from  Battle  Command  graphic  TP  525-3.3  and  leaders  cluster  in  concept  map) 

—  Agility,  as  it  applies  to  joint  C2,  has  six  key  elements:  robustness,  resilience,  adaptability, 
responsiveness,  flexibility,  and  innovation  (TP  525-3.3) 

Plan  adjustment  during  mission  execution  is  a  form  of  leader  flexibility  (inference 
from  and  lead  cluster) 

—  No  plan  survives  intact  once  contact  is  made.  Tactical  flexibility  requires  flexible  leaders  capable  of 
adapting  to  rapidly  changing  circumstances;  and  staffs  able  to  recognize  significant  changes  in  the 
situation,  and  resynchronize  the  operation  by  coordinating  the  changes  to  alter  the  plan  (FM  6-0). 

Agility  &  flexibility  depend  on  (semantic  &  pragmatic)  interoperability 

—  in  order  to  share  needed  information  when  it  is  needed  &  in  a  form  it  can  be  understood  and  acted  on 
with  confidence  (TP  525-3.3). 


Agility,  flexibility  &  interoperability  are  battle  command  quality  attributes 

•  need  to  be  operationalized  with  numeric  thresholds  and  objectives  specified 

•  Planning  &  plan  reframing  provide  context  for  specifying  these  measures. 
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Iterations  =  1000 


Planning  Aviation  Missions  (PAM) 
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Interpreting  the  Significance  of  Adjust  Plan  in  the 

PAM  DFD 


Statements  in  the  document  do  not  explicitly  commit  to  plan  adjustments  in-flight. 
While  the  document  says 

-  “The  ...  interface  to  the  Maneuver  Control  System  (MCS)  ...  provides  the  aviation 
commander  with  continuous  updates  of  the  friendly  and  enemy  situation  and  allows  the 
commander  to  rapidly  adjust  his  plan  to  accomplish  his  assigned  mission.  ” 

-  it  also  says,  “The  assigned  missions,  orders  and  map  data  are  then  transferred  down 
to  the  air  crew  level  where  specific  air  crew  mission  planning  takes  place.  This  mission 
information  is  then  loaded  into  the  aircraft  systems  via  hardware  or  digital  radio  transfer 
for  use  during  mission  execution.  ” 

-  In  fact,  in  conversation  with  aviation  mission  planning  acquisition  people,  they 
asserted  their  system  only  served  pre-flight  planning. 

High  level  interoperability  requirements  are  stated,  but  not  clearly  defined  or 
operationalized  with  measures  specified. 
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Iterations  =  1000 


[Minion  | 


IconsuWierl 

Performan^_AttriDutes 

sendingsyfetem 
receivtngj;system 


/ 


^cf$ate 

A 

approval 

/ 

/  ( 

Driers 

area 

^btnoving 

engmy 

software 


Interoperability  in 
ArrriyA^lation  ISP 
for  Mission 
Planning 

^"Tra^iing 


receive 


assess  r. 


situation  . 


ions 


aircraft 


wether 

\  - / 


planning 


Software  Engineering  Institute  CamegieMellon 


access 


An  Innovative  Requirements  Solution:  Marrying 
Six  Sigma  KJ  Analysis  with  Automation  for  Text 
Analysis  and  Collaboration  |  SEPG  2009  North 
America 

©  2009  Carnegie  Mellon  University 


Interpreting  the  Significance  of  Interoperability  in 

the  PAM  ISP 


According  to  the  ISP,  most  of  the  interoperability  functions  support  data 
transfer  in  various  formats  enabling  the  planning  system  to  be 

—  the  collector  &  consolidator  of  all  pertinent  battlefield  information  needed  for 
effective  aviation  missions  thereby  serving  as  an  information  consumer 

—  capable  of  in-flight  re-targeting  and  re-planning  (though  this  is  only  stated 
once  as  an  objective) 

However,  supporting  data  transfer  in  various  formats  is  syntactic  interoperability  - 
not  semantic  or  pragmatic  interoperability  needed  for  in-flight  re-targeting  &  re¬ 
planning. 

Interoperability  levels  labeled  0,  1  or  2  are  used  but  not  defined,  let  alone 
operationalized  with  measures  specified. 
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